[FLINK-34913] ConcurrentModificationException SubTaskInitializationMetricsBuilder.addDurationMetric

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.19.0
• Fix Version/s: 1.20.0
• Component/s: Runtime / Metrics, Runtime / State Backends
• Labels: pull-request-available
• Assignee: Piotr Nowojski
• Reporter: Piotr Nowojski
• Created: 22/Mar/24 08:14
• Updated: 23/Apr/24 07:47
• Resolved: 23/Apr/24 07:47

Description

The following failures can occur during job's recovery when using clip & ingest. This code path is currently not available, so the bug can not happen for users.

java.util.ConcurrentModificationException
	at java.base/java.util.HashMap.compute(HashMap.java:1230)
	at org.apache.flink.runtime.checkpoint.SubTaskInitializationMetricsBuilder.addDurationMetric(SubTaskInitializationMetricsBuilder.java:49)
	at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.runAndReportDuration(RocksDBIncrementalRestoreOperation.java:988)
	at org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.lambda$createAsyncCompactionTask$2(RocksDBIncrementalRestoreOperation.java:291)

java.util.ConcurrentModificationException
	at java.base/java.util.HashMap.compute(HashMap.java:1230)
	at org.apache.flink.runtime.checkpoint.SubTaskInitializationMetricsBuilder.addDurationMetric(SubTaskInitializationMetricsBuilder.java:49)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreStateAndGates(StreamTask.java:794)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$restoreInternal$3(StreamTask.java:744)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:744)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:704)
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953)
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:922)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
	at java.base/java.lang.Thread.run(Thread.java:829)

Issue Links

• is caused by: FLINK-34199 Add tracing for durations of rescaling/restoring RocksDB incremental checkpoints from downloaded and local state
• links to: GitHub Pull Request #24554