[SPARK-48544] Reduce memory pressure of empty TreeNode BitSets

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Optimizer
• Labels: pull-request-available
• Assignee: Nick Young
• Reporter: Nick Young
• Created: 05/Jun/24 20:36
• Updated: 10/Jun/24 18:19
• Resolved: 10/Jun/24 18:18

Description

By default, the TreeNode `treePatternBits` and `ineffectiveRules` variables are initialized to a `BitSet`, which takes up memory even when empty. For plans with a high number of TreeNodes, this adds unnecessary memory pressure, especially for large expression trees. Reduce the memory pressure by lazily initializing this variable.

For `treePatternBits` we will observe no benefit because these are always populated. However, for `ineffectiveRules` it is not necessary to initialize this for every node. We should make this variable lazy, being careful to not initialize even when `get` is called.

Related to SPARK-48047.

Issue Links

• links to: GitHub Pull Request #46919