[SPARK-45205] Since version 3.2.0, Spark SQL has taken longer to execute "show paritions",probably because of changes introduced by SPARK-35278

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 4.0.0, 3.5.1
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Qiang Yang
• Reporter: Qiang Yang
• Created: 18/Sep/23 16:58
• Updated: 10/Oct/23 06:38
• Resolved: 10/Oct/23 06:37

Description

After SPARK-35378 was changed, I noticed that the execution of statements such as ‘show parititions test' became slower.

The change point is that the execution process changes from ExecutedCommandEnec to CommandResultExec, but ExecutedCommandExec originally implemented the following method

override def executeToIterator(): Iterator[InternalRow] = sideEffectResult.iterator

CommandResultExec is not rewritten, so when the hasNext method is executed, a job process is created, resulting in increased time-consuming

Issue Links

• links to: GitHub Pull Request #43270