[SPARK-41651] Test parity: pyspark.sql.tests.test_dataframe

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Done
• Affects Version/s: 3.4.0
• Component/s: Connect
• Epic Link: Spark Connect
• Assignee: Sandeep Singh
• Reporter: Hyukjin Kwon
• Created: 21/Dec/22 07:16
• Updated: 10/Jul/23 02:22
• Resolved: 10/Jul/23 02:22

Description

After https://github.com/apache/spark/pull/39041 (SPARK-41528), we now reuses the same test cases, see python/pyspark/sql/tests/connect/test_parity_dataframe.py.

We should remove all the test cases defined there, and fix Spark Connect behaviours accordingly.

Issue Links

• is part of: SPARK-41285 Test basework and improvement of test coverage in Spark Connect

Sub-Tasks

• Enable test_df_show, test_drop, test_dropna, test_toDF_with_schema_string and test_with_columns_renamed: Resolved
• Enable test_freqItems, test_input_files, test_toDF_with_schema_string, test_to_pandas_required_pandas_not_found: Resolved
• Add JIRAs or messages for skipped tests: Resolved
• Enable test_generic_hints: Resolved