[SPARK-35355] improve execution performance in insert...select...limit case

• Type: Improvement
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: yikaifei
• Created: 09/May/21 06:19
• Updated: 16/Dec/21 02:48

Description

In the case of `insert into...select...limit` , `CollectLimitExec` has better execution performance than `GlobalLimit` .

Before:

== Physical Plan ==
 Execute InsertIntoHadoopFsRelationCommand ...
 +- *(2) GlobalLimit 5
 +- Exchange SinglePartition, true, id=#39
 +- *(1) LocalLimit 5
 +- *(1) ColumnarToRow
 +- FileScan ...

After:

== Physical Plan ==
 Execute InsertIntoHadoopFsRelationCommand ...
 +- CollectLimit 5
 +- *(1) ColumnarToRow
 +- FileScan ....

Issue Links

• links to: [Github] Pull Request #32483 (yikf)
• links to: [Github] Pull Request #34916 (yikf)