[SPARK-47445] Adding new 'Silent' ExplainMode

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Connect, Documentation, PySpark, SQL
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Victor Sunderland
• Created: 18/Mar/24 16:49
• Updated: 18/Mar/24 22:59

Description

While investigating unit test duration we found that org.apache.spark.sql.execution.QueryExecution.explainString () takes approximately 14% time. This method generates the string representation of the execution plan. The string is often used for logging purposes. This is also called for each AQE job so it can save prod execution time too. While SPARK-44485 does exist to help optimize the prod execution time, the main purpose of this PR is to save time during unit testing.

I've added a silent mode to ExplainMode to try and mitigate this issue.

Issue Links

• links to: GitHub Pull Request #45542