[SPARK-27893] Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

• Type: Test
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: PySpark, SQL
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 31/May/19 04:02
• Updated: 12/Dec/22 18:10
• Resolved: 03/Jun/19 01:04

Description

See https://github.com/apache/spark/pull/24675#issuecomment-495591310

Due to lack of Python tests, and due to difficulty about writing a PySpark tests for plans, we have faced multiple issues so far.

This JIRA targets to add a test base that works with SQL files.

Issue Links

• is related to: SPARK-28359 Make integrated UDF tests robust by making them no-op
• is related to: SPARK-28536 Reduce shuffle partitions in Python UDF tests in SQLQueryTestSuite
• is related to: SPARK-27921 Convert applicable *.sql tests into UDF integrated test base
• links to: GitHub Pull Request #24752
• links to: GitHub Pull Request #24945