[SPARK-46291] Koalas Testing Migration

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.2.0
• Component/s: PySpark
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Created: 06/Dec/23 21:55
• Updated: 06/Dec/23 22:03
• Resolved: 06/Dec/23 22:01

Description

Test migration from Koalas to Spark repository, including setting up the testing environment and dependencies, and CI jobs.

Issue Links

• is part of: SPARK-34849 SPIP: Support pandas API layer on PySpark

Sub-Tasks

• Port/integrate Koalas DataFrame unit test into PySpark: Resolved
• Port/integrate Koalas dependencies into PySpark: Resolved
• Port Koalas Series related unit tests into PySpark: Resolved
• Consolidate PySpark testing utils: Resolved
• Port Koalas DataFrame related unit tests into PySpark: Resolved
• Port Koalas operations on different frames tests into PySpark: Resolved
• Port Koalas Index unit tests into PySpark: Resolved
• Revisit pandas-on-Spark test cases that are disabled because of pandas nondeterministic return values: Resolved
• Port Koalas plot unit tests into PySpark: Resolved
• Port Koalas miscellaneous unit tests into PySpark: Resolved
• Port Koalas internal implementation unit tests into PySpark: Resolved
• Remove Spark-version related codes from test codes.: Resolved
• Standardize module name in install.rst: Resolved