[FLINK-19012] E2E test fails with "Cannot register Closeable, this subtaskCheckpointCoordinator is already closed. Closing argument."

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.12.0
• Component/s: Runtime / Checkpointing, Runtime / Task, Tests
• Labels: pull-request-available test-stability
• Assignee: Roman Khachatryan
• Reporter: Robert Metzger
• Created: 21/Aug/20 07:41
• Updated: 03/Sep/20 13:26
• Resolved: 03/Sep/20 13:26

Description

Note: This error occurred in a custom branch with unreviewed changes. I don't believe my changes affect this error, but I would keep this in mind when investigating the error: https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8307&view=logs&j=1f3ed471-1849-5d3c-a34c-19792af4ad16&t=0d2e35fc-a330-5cf2-a012-7267e2667b1d

2020-08-20T20:55:30.2400645Z 2020-08-20 20:55:22,373 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Registering task at network: Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1) (cbc357ccb763df2852fee8c4fc7d55f2_0_0) [DEPLOYING].
2020-08-20T20:55:30.2402392Z 2020-08-20 20:55:22,401 INFO  org.apache.flink.streaming.runtime.tasks.StreamTask          [] - No state backend has been configured, using default (Memory / JobManager) MemoryStateBackend (data in heap memory / checkpoints to JobManager) (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize: 5242880)
2020-08-20T20:55:30.2404297Z 2020-08-20 20:55:22,413 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1) (cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from DEPLOYING to RUNNING.
2020-08-20T20:55:30.2405805Z 2020-08-20 20:55:22,786 INFO  org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6ApiCallBridge [] - Pinging Elasticsearch cluster via hosts [http://127.0.0.1:9200] ...
2020-08-20T20:55:30.2407027Z 2020-08-20 20:55:22,848 INFO  org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6ApiCallBridge [] - Elasticsearch RestHighLevelClient is connected to [http://127.0.0.1:9200]
2020-08-20T20:55:30.2409277Z 2020-08-20 20:55:29,205 INFO  org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorImpl [] - Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1) discarding 0 drained requests
2020-08-20T20:55:30.2410690Z 2020-08-20 20:55:29,218 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1) (cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from RUNNING to FINISHED.
2020-08-20T20:55:30.2412187Z 2020-08-20 20:55:29,218 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Freeing task resources for Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1) (cbc357ccb763df2852fee8c4fc7d55f2_0_0).
2020-08-20T20:55:30.2414203Z 2020-08-20 20:55:29,224 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - Un-registering task and sending final execution state FINISHED to JobManager for task Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1) cbc357ccb763df2852fee8c4fc7d55f2_0_0.
2020-08-20T20:55:30.2415602Z 2020-08-20 20:55:29,219 INFO  org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - Source: Sequence Source -> Flat Map -> Sink: Unnamed (1/1) - asynchronous part of checkpoint 1 could not be completed.
2020-08-20T20:55:30.2416411Z java.io.UncheckedIOException: java.io.IOException: Cannot register Closeable, this subtaskCheckpointCoordinator is already closed. Closing argument.
2020-08-20T20:55:30.2418956Z 	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.lambda$registerConsumer$2(SubtaskCheckpointCoordinatorImpl.java:468) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2020-08-20T20:55:30.2420100Z 	at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:91) [flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2020-08-20T20:55:30.2420927Z 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_265]
2020-08-20T20:55:30.2421455Z 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_265]
2020-08-20T20:55:30.2421879Z 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_265]
2020-08-20T20:55:30.2422348Z Caused by: java.io.IOException: Cannot register Closeable, this subtaskCheckpointCoordinator is already closed. Closing argument.
2020-08-20T20:55:30.2423416Z 	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.registerAsyncCheckpointRunnable(SubtaskCheckpointCoordinatorImpl.java:378) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2020-08-20T20:55:30.2424635Z 	at org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.lambda$registerConsumer$2(SubtaskCheckpointCoordinatorImpl.java:466) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
2020-08-20T20:55:30.2425174Z 	... 4 more
2020-08-20T20:55:30.2426945Z 2020-08-20 20:55:29,339 INFO  org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl [] - Free slot TaskSlot(index:0, state:ACTIVE, resource profile: ResourceProfile{cpuCores=1.0000000000000000, taskHeapMemory=384.000mb (402653174 bytes), taskOffHeapMemory=0 bytes, managedMemory=512.000mb (536870920 bytes), networkMemory=128.000mb (134217730 bytes)}, allocationId: f5938e3baed0f9564aa17169f68947bd, jobId: 333b1654bd93574471bd44ad45847379).
2020-08-20T20:55:30.2428701Z 2020-08-20 20:55:29,354 INFO  org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Remove job 333b1654bd93574471bd44ad45847379 from job leader monitoring.

Issue Links

• is related to: FLINK-19093 "Elasticsearch (v6.3.1) sink end-to-end test" failed with "SubtaskCheckpointCoordinatorImpl was closed without closing asyncCheckpointRunnable 1"
• links to: GitHub Pull Request #13267