[SPARK-32332] AQE doesn't adequately allow for Columnar Processing extension

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.1, 3.1.0
• Component/s: SQL
• Assignee: Wenchen Fan
• Reporter: Thomas Graves
• Created: 16/Jul/20 14:50
• Updated: 31/Jul/20 16:15
• Resolved: 29/Jul/20 19:25

Description

In SPARK-27396 we added support to extended Columnar Processing. We did the initial work as to what we thought was sufficient but adaptive query execution was being developed at the same time.

We have discovered that the changes made to AQE are not sufficient for users to properly extend it for columnar processing because AQE hardcodes to look for specific classes/execs.

 

Issue Links

• is related to: SPARK-32430 Allow plugins to inject rules into AQE query stage preparation
• links to: [Github] Pull Request #29134 (andygrove)
• links to: [Github] Pull Request #29134 (andygrove)
• links to: [Github] Pull Request #29262 (cloud-fan)
• links to: [Github] Pull Request #29262 (cloud-fan)
• links to: [Github] Pull Request #29310 (andygrove)