[SPARK-35042] Support traversal pruning in transform/resolve functions and their call sites - ASF JIRA

Attach files

Attach Screenshot

Add vote

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: 3.2.0
Fix Version/s: None
Component/s: Optimizer
Labels:
None

Description

Transform/resolve functions are called ~280k times per query on average for a TPC-DS query, which are way more than necessary. We can reduce those calls with early exit information and conditions. This doc some evaluation numbers with a prototype.

Attachments

Sub-Tasks

Create Sub-Task

1.	Support traversal pruning in the transform function family	Resolved	Yingyi Bu	Actions
2.	Support traversal pruning in resolve functions in AnalysisHelper	Resolved	Yingyi Bu	Actions
3.	Use static treePatternBitSet for Leaf expressions like AttributeReference and Literal	Resolved	Yingyi Bu	Actions
4.	Fix BitSet.union	Resolved	Unassigned	Actions
5.	Migrate to transformWithPruning or resolveWithPruning for subquery related rules	Resolved	Yingyi Bu	Actions
6.	Migrate to transformWithPruning for leftover optimizer rules	Resolved	Yingyi Bu	Actions
7.	Migrate to transformWithPruning or resolveWithPruning for expression rules	Resolved	Yingyi Bu	Actions
8.	Migrate to transformWithPruning or resolveWithPruning for object rules	Resolved	Yingyi Bu	Actions
9.	Migrate to transformWithPruning or resolveWithPruning for rules in finishAnalysis	Resolved	Yingyi Bu	Actions
10.	Migrate to resolveWithPruning for two command rules	Resolved	Unassigned	Actions
11.	Support traversal pruning in transformUpWithNewOutput	Open	Unassigned	Actions
12.	Add rule id to all Analyzer rules in fixed point batches	Resolved	Yingyi Bu	Actions
13.	Migrate to transformWithPruning for top-level rules under catalyst/optimizer	Resolved	Yingyi Bu	Actions
14.	Migrate to transformWithPruning for rules in optimizer/Optimizer.scala	Resolved	Yingyi Bu	Actions
15.	Migrate transformAllExpressions callsites to transformAllExpressionsWithPruning	Resolved	Apache Spark	Actions
16.	Add tree pattern pruning into Analyzer rules	Resolved	Yingyi Bu	Actions
17.	Add rule id pruning to the TypeCoercion rule	Resolved	Yingyi Bu	Actions
18.	Support traversal pruning in extendedResolutionRules and postHocResolutionRules	In Progress	Unassigned	Actions
19.	Add tree pattern pruning to CTESubstitution rule	Resolved	Josh Rosen	Actions
20.	Identify aggregation expression in the nodePatterns of PythonUDF	Resolved	Gengliang Wang	Actions
21.	Add a linter rule to enforce transforming with pruning	Open	Unassigned	Actions