[FLINK-33184] HybridShuffleITCase fails with exception in resource cleanup of task Map on AZP

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.19.0
• Component/s: Runtime / Network
• Labels: test-stability
• Assignee: Unassigned
• Reporter: Sergey Nuyanzin
• Created: 04/Oct/23 16:53
• Updated: 25/Jan/24 03:56
• Resolved: 24/Nov/23 02:38

Description

This build fails https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53548&view=logs&j=baf26b34-3c6a-54e8-f93f-cf269b32f802&t=8c9d126d-57d2-5a9e-a8c8-ff53f7b35cd9&l=8710

 
Map (5/10)#0] ERROR org.apache.flink.runtime.taskmanager.Task                    [] - FATAL - exception in resource cleanup of task Map (5/10)#0 (159f887fbd200ea7cfa4aaeb1127c4ab_0a448493b4782967b150582570326227_4_0)
.
java.lang.IllegalStateException: Leaking buffers.
        at org.apache.flink.util.Preconditions.checkState(Preconditions.java:193) ~[flink-core-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.TieredStorageMemoryManagerImpl.release(TieredStorageMemoryManagerImpl.java:236) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_292]
        at org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.TieredStorageResourceRegistry.clearResourceFor(TieredStorageResourceRegistry.java:59) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.io.network.partition.hybrid.tiered.shuffle.TieredResultPartition.releaseInternal(TieredResultPartition.java:195) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.io.network.partition.ResultPartition.release(ResultPartition.java:262) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.io.network.partition.ResultPartitionManager.releasePartition(ResultPartitionManager.java:88) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.io.network.partition.ResultPartition.fail(ResultPartition.java:284) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.taskmanager.Task.failAllResultPartitions(Task.java:1004) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.taskmanager.Task.releaseResources(Task.java:990) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:838) [flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) [flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
01:17:22,375 [flink-pekko.actor.default-dispatcher-5] INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Task Sink: Unnamed (3/10)#0 is already in state CANCELING
01:17:22,375 [        Map (5/10)#0] ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor           [] - FATAL - exception in resource cleanup of task Map (5/10)#0 (159f887fbd200ea7cfa4aaeb1127c4ab_0a448493b4782967b150582570326227_4_0)
.
java.lang.IllegalStateException: Leaking buffers.
        at org.apache.flink.util.Preconditions.checkState(Preconditions.java:193) ~[flink-core-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.TieredStorageMemoryManagerImpl.release(TieredStorageMemoryManagerImpl.java:236) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_292]
        at org.apache.flink.runtime.io.network.partition.hybrid.tiered.storage.TieredStorageResourceRegistry.clearResourceFor(TieredStorageResourceRegistry.java:59) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.io.network.partition.hybrid.tiered.shuffle.TieredResultPartition.releaseInternal(TieredResultPartition.java:195) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.io.network.partition.ResultPartition.release(ResultPartition.java:262) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.io.network.partition.ResultPartitionManager.releasePartition(ResultPartitionManager.java:88) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.io.network.partition.ResultPartition.fail(ResultPartition.java:284) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.taskmanager.Task.failAllResultPartitions(Task.java:1004) ~[flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.taskmanager.Task.releaseResources(Task.java:990) [flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:838) [flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) [flink-runtime-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]

Issue Links

• is fixed by: FLINK-33185 HybridShuffleITCase fails with TimeoutException: Pending slot request timed out in slot pool on AZP