[SPARK-36666] [SQL] Regression in AQEShuffleReadExec

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Andy Grove
• Reporter: Andy Grove
• Shepherd: Thomas Graves
• Created: 03/Sep/21 23:51
• Updated: 07/Sep/21 20:50
• Resolved: 07/Sep/21 20:50

Description

I am currently testing the RAPIDS Accelerator for Apache Spark with the Spark 3.2 release candidate and there is a regression in AQEShuffleReadExec where it now throws an exception if the shuffle's output partitioning does not match a specific list of schemes.

The problem can be solved by returning UnknownPartitioning, as it already does in some cases, rather than throwing an exception.

Issue Links

• relates to: SPARK-36315 Only skip AQEShuffleReadRule in the final stage if it breaks the distribution requirement
• links to: [Github] Pull Request #33910 (andygrove)