[SPARK-40508] Treat unknown partitioning as UnknownPartitioning

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: Spark Core
• Assignee: Ted Yu
• Reporter: Ted Yu
• Created: 20/Sep/22 20:08
• Updated: 28/Sep/22 03:35
• Resolved: 21/Sep/22 16:41

Description

When running spark application against spark 3.3, I see the following :

java.lang.IllegalArgumentException: Unsupported data source V2 partitioning type: CustomPartitioning
    at org.apache.spark.sql.execution.datasources.v2.V2ScanPartitioning$$anonfun$apply$1.applyOrElse(V2ScanPartitioning.scala:46)
    at org.apache.spark.sql.execution.datasources.v2.V2ScanPartitioning$$anonfun$apply$1.applyOrElse(V2ScanPartitioning.scala:34)
    at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)

The CustomPartitioning works fine with Spark 3.2.1
This PR proposes to relax the code and treat all unknown partitioning the same way as that for UnknownPartitioning.

Issue Links

• is related to: SPARK-37375 Umbrella: Storage Partitioned Join (SPJ)
• links to: [Github] Pull Request #37952 (tedyu)
• links to: [Github] Pull Request #37957 (tedyu)
• links to: [Github] Pull Request #37957 (tedyu)