[FLINK-22458] Failed when operating empty catalog table

• Type: Bug
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: 1.12.2
• Component/s: Connectors / Hive
• Labels: auto-deprioritized-major auto-deprioritized-minor

Environment

Flink: 1.12.2

• Assignee: Unassigned
• Reporter: Spongebob
• Created: 26/Apr/21 03:03
• Updated: 03/Jan/22 10:39

Description

The pipline might like: HiveTable -> FlinkCatalogTable(might be empty) -> HiveTable

It runs normally when the FlinkCatalogTable is not empty, But When FlinkCatalogTable is empty, Jobmanager throws this exception:

java.lang.Exception: Failed to finalize execution on master at org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1373) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.ExecutionVertex.executionFinished(ExecutionVertex.java:877) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.Execution.markFinished(Execution.java:1241) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.ExecutionGraph.updateStateInternal(ExecutionGraph.java:1610) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.ExecutionGraph.updateState(ExecutionGraph.java:1584) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:663) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:447) ~[flownData-1.0-jar-with-dependencies.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_251] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_251] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flownData-1.0-jar-with-dependencies.jar:?] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flownData-1.0-jar-with-dependencies.jar:?] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flownData-1.0-jar-with-dependencies.jar:?] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flownData-1.0-jar-with-dependencies.jar:?] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flownData-1.0-jar-with-dependencies.jar:?] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flownData-1.0-jar-with-dependencies.jar:?] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flownData-1.0-jar-with-dependencies.jar:?] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flownData-1.0-jar-with-dependencies.jar:?] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flownData-1.0-jar-with-dependencies.jar:?] at akka.actor.Actor.aroundReceive(Actor.scala:517) [flownData-1.0-jar-with-dependencies.jar:?] at akka.actor.Actor.aroundReceive$(Actor.scala:515) [flownData-1.0-jar-with-dependencies.jar:?] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flownData-1.0-jar-with-dependencies.jar:?] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flownData-1.0-jar-with-dependencies.jar:?] at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flownData-1.0-jar-with-dependencies.jar:?] Caused by: org.apache.flink.table.api.TableException: Exception in finalizeGlobal at org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:94) ~[flink-table-blink_2.12-1.12.2.jar:1.12.2] at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:148) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1368) ~[flownData-1.0-jar-with-dependencies.jar:?] ... 35 more Caused by: java.io.FileNotFoundException: File hdfs://hadmaster:9898/data/hive_warehouse/ticket_data.db/dwd_flown_fin_segment/.staging_1619403238924 does not exist. at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1053) ~[hadoop-hdfs-client-3.1.3.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131) ~[hadoop-hdfs-client-3.1.3.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1113) ~[hadoop-hdfs-client-3.1.3.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1110) ~[hadoop-hdfs-client-3.1.3.jar:?] at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.1.3.jar:?] at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1120) ~[hadoop-hdfs-client-3.1.3.jar:?] at org.apache.flink.hive.shaded.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:170) ~[flink-connector-hive_2.12-1.12.2.jar:1.12.2] at org.apache.flink.table.filesystem.PartitionTempFileManager.headCheckpoints(PartitionTempFileManager.java:137) ~[flink-table-blink_2.12-1.12.2.jar:1.12.2] at org.apache.flink.table.filesystem.FileSystemCommitter.commitUpToCheckpoint(FileSystemCommitter.java:93) ~[flink-table-blink_2.12-1.12.2.jar:1.12.2] at org.apache.flink.table.filesystem.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:92) ~[flink-table-blink_2.12-1.12.2.jar:1.12.2] at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:148) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1368) ~[flownData-1.0-jar-with-dependencies.jar:?] ... 35 more 2021-04-26 10:14:31,726 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job insert-into_hive.ticket_data.dwd_flown_fin_segment (5e2132be01c51bb5279ed535b730448b) switched from state RUNNING to FAILING.
org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:118) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getGlobalFailureHandlingResult(ExecutionFailureHandler.java:93) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.scheduler.DefaultScheduler.handleGlobalFailure(DefaultScheduler.java:252) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.scheduler.UpdateSchedulerNgOnInternalFailuresListener.notifyGlobalFailure(UpdateSchedulerNgOnInternalFailuresListener.java:62) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.ExecutionGraph.failGlobal(ExecutionGraph.java:1116) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.ExecutionGraph.vertexFinished(ExecutionGraph.java:1373) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.ExecutionVertex.executionFinished(ExecutionVertex.java:877) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.Execution.markFinished(Execution.java:1241) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.ExecutionGraph.updateStateInternal(ExecutionGraph.java:1610) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.executiongraph.ExecutionGraph.updateState(ExecutionGraph.java:1584) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:663) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:89) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:447) ~[flownData-1.0-jar-with-dependencies.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_251] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_251] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) ~[flownData-1.0-jar-with-dependencies.jar:?] at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158) ~[flownData-1.0-jar-with-dependencies.jar:?] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) [flownData-1.0-jar-with-dependencies.jar:?] at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) [flownData-1.0-jar-with-dependencies.jar:?] at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) [flownData-1.0-jar-with-dependencies.jar:?] at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) [flownData-1.0-jar-with-dependencies.jar:?] at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) [flownData-1.0-jar-with-dependencies.jar:?] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) [flownData-1.0-jar-with-dependencies.jar:?] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flownData-1.0-jar-with-dependencies.jar:?] at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) [flownData-1.0-jar-with-dependencies.jar:?] at akka.actor.Actor.aroundReceive(Actor.scala:517) [flownData-1.0-jar-with-dependencies.jar:?] at akka.actor.Actor.aroundReceive$(Actor.scala:515) [flownData-1.0-jar-with-dependencies.jar:?] at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) [flownData-1.0-jar-with-dependencies.jar:?] at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) [flownData-1.0-jar-with-dependencies.jar:?] at akka.actor.ActorCell.invoke(ActorCell.scala:561) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.Mailbox.run(Mailbox.scala:225) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.Mailbox.exec(Mailbox.scala:235) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [flownData-1.0-jar-with-dependencies.jar:?] at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [flownData-1.0-jar-with-dependencies.jar:?]

Issue Links

• relates to: FLINK-20784 .staging_xxx does not exist, when insert into hive