[SPARK-44485] optimize generateTreeString code path

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.1
• Fix Version/s: 3.5.0, 4.0.0
• Component/s: Spark Core
• Assignee: Ziqi Liu
• Reporter: Ziqi Liu
• Created: 19/Jul/23 21:25
• Updated: 22/Jul/23 07:44
• Resolved: 22/Jul/23 07:44

Description

In `TreeNode.generateTreeString`, we observed inefficiency in scala collection operations and virtual function call.

This inefficiency become significant in large plan (we hit a example of more than 1000 nodes). So it’s worth optimizing the super hot code path. By rewriting into native Java code(not so sweet as scala syntax sugar though), we should be able to get rid of most of the overhead.