Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28359

Make integrated UDF tests robust by making them no-op

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • PySpark, SQL
    • None

    Description

      Current UDFs available in `IntegratedUDFTestUtils` are not exactly no-op. It converts input column to strings and outputs to strings.

      This causes many issues, for instance, https://github.com/apache/spark/pull/25128 or https://github.com/apache/spark/pull/25110

      Ideally we should make this UDF virtually noop.

      Attachments

        Issue Links

          relates to

          Test - A new unit, integration or system test. SPARK-27893 Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Umbrella - An overarching type made of sub-tasks SPARK-27921 Convert applicable *.sql tests into UDF integrated test base

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #25130

          Web Link GitHub Pull Request #25132

          Activity

            People

              gurwls223 Hyukjin Kwon
              gurwls223 Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: