[SPARK-46041] Increase Test Coverage

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: PySpark, Tests
• Assignee: Unassigned
• Reporter: Xinrong Meng
• Shepherd: Xinrong Meng
• Created: 21/Nov/23 23:51
• Updated: 14/Dec/23 21:50

Description

Our current PySpark test coverage stands at approximately 80% as reported on Codecov, which shows a recent decrease in test coverage. It is essential that we make efforts to enhance our test coverage going forward.

Our objective is to significantly increase our PySpark test coverage to reach >90%.

Issue Links

• Testing discovered: SPARK-46277 Validate startup urls with the config being set
• Testing discovered: SPARK-46386 Improve assertions of observation (pyspark.sql.observation)
• Testing discovered: SPARK-46413 Validate returnType of Arrow Python UDF

Sub-Tasks

• Improve test coverage of udf.py: Resolved
• Test invalid error class (pyspark.errors.utils): Resolved
• Test SparkPandasNotImplementedError (pyspark.pandas.exceptions): Resolved
• Test string representation of TestResult (pyspark.mllib.stat.test): Resolved
• Improve test coverage of memory_profiler.py: Resolved
• Improve test coverage of pyspark utils: Resolved
• Test captured errors (pyspark.errors.exceptions.captured): Resolved
• Exclude generated files from the code coverage report: Resolved
• Test catalog error classes (pyspark.sql.catalog): Resolved
• Test missing test coverage for Column (pyspark.sql.column): Resolved
• Test invalid key for spark.conf.get (pyspark.sql.conf): Resolved
• Test missing cases for SparkSession (pyspark.sql.session): Resolved
• Exclude ported loose_version from the code coverage report: Resolved
• Test missing cases for functions (pyspark.sql.functions): Resolved
• Test aggregate functions for groups (pyspark.sql.group): Resolved
• Test rangeBetween window function (pyspark.sql.window): Resolved
• Improve and test exceptions of TimedeltaIndex: Resolved
• Test custom data source and input partition (pyspark.sql.datasource): Resolved
• Remove unnecessary dependency from test_mixed_udf_and_sql.py: Resolved
• Improve and test warning for Arrow-optimized Python UDF: Resolved