[SPARK-32838] Connot insert overwite different partition with same table

• Type: Sub-task
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: SQL

Environment

hadoop 2.7 + spark 3.0.0

• Assignee: Unassigned
• Reporter: CHC
• Created: 10/Sep/20 03:54
• Updated: 05/Apr/22 06:17

Description

When:

CREATE TABLE tmp.spark3_snap (
id string
)
PARTITIONED BY (dt string)
STORED AS ORC
;

insert overwrite table tmp.spark3_snap partition(dt='2020-09-09')
select 10;
insert overwrite table tmp.spark3_snap partition(dt='2020-09-10')
select 1;

insert overwrite table tmp.spark3_snap partition(dt='2020-09-10')
select id from tmp.spark3_snap where dt='2020-09-09';

and it will be get a error: "Cannot overwrite a path that is also being read from"

related: https://issues.apache.org/jira/browse/SPARK-24194

This work on spark 2.4.3 and do not work on spark 3.0.0
 

Issue Links

• is related to: SPARK-33144 Connot insert overwite multiple partition, get exception "get partition: Value for key name is null or empty"
• relates to: SPARK-24194 HadoopFsRelation cannot overwrite a path that is also being read from
• links to: [Github] Pull Request #30057 (AngersZhuuuu)
• links to: [Github] Pull Request #35608 (AngersZhuuuu)
• links to: [Github] Pull Request #36070 (CHENXCHEN)
• links to: [Github] Pull Request #36070 (CHENXCHEN)