[FLINK-25049] sql-client.sh execute batch job in async mode failed with java.io.FileNotFoundException

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.14.0
• Component/s: Connectors / Hive, Table SQL / Ecosystem
• Assignee: Unassigned
• Reporter: macdoor615
• Created: 25/Nov/21 06:44
• Updated: 30/Nov/21 13:09

Description

execute multi simple sql in a sql file, like 

 

insert overwrite bnpmp.p_biz_hcd_5m select * from bnpmp.p_biz_hcd_5m where dt='2021-11-22';
insert overwrite bnpmp.p_biz_hjr_5m select * from bnpmp.p_biz_hjr_5m where dt='2021-11-22';
insert overwrite bnpmp.p_biz_hswtv_5m select * from bnpmp.p_biz_hswtv_5m where dt='2021-11-22';
...

 

if 

SET table.dml-sync = true;

execute properly.

if

SET table.dml-sync = false;

some SQL Job failed with the following error
 

 Caused by: java.lang.Exception: Failed to finalize execution on master ... 37 more Caused by: org.apache.flink.table.api.TableException: Exception in finalizeGlobal at org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:91) at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:148) at org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.vertexFinished(DefaultExecutionGraph.java:1086) ... 36 more Caused by: java.io.FileNotFoundException: File hdfs://service1/user/hive/warehouse/bnpmp.db/p_snmp_f5_traffic_5m/.staging_1637821441497 does not exist. at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:901) at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:112) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:961) at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:958) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:958) at org.apache.flink.hive.shaded.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:170) at org.apache.flink.table.filesystem.PartitionTempFileManager.listTaskTemporaryPaths(PartitionTempFileManager.java:104) at org.apache.flink.table.filesystem.FileSystemCommitter.commitPartitions(FileSystemCommitter.java:77) at org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:89) ... 38 more