[SPARK-48817] MultiInsert is split to multiple sql executions, resulting in no exchange reuse

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0, 3.5.1
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Zhen Wang
• Reporter: Zhen Wang
• Created: 05/Jul/24 06:58
• Updated: 10/Jul/24 12:26
• Resolved: 10/Jul/24 05:10

Description

MultiInsert is split to multiple sql executions, resulting in no exchange reuse.

 

Reproduce sql:

create table wangzhen_t1(c1 int);
create table wangzhen_t2(c1 int);
create table wangzhen_t3(c1 int);

insert into wangzhen_t1 values (1), (2), (3);

from (select /*+ REPARTITION(3) */ c1 from wangzhen_t1)
insert overwrite table wangzhen_t2 select c1
insert overwrite table wangzhen_t3 select c1; 

 

In Spark 3.1, there is only one SQL execution and there is a reuse exchange.

 

However, in Spark 3.5, it was split to multiple executions and there was no ReuseExchange.

 

Issue Links

• links to: GitHub Pull Request #47224