[FLINK-28789] TPC-DS tests failed due to release input gate for task failure

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.16.0
• Fix Version/s: 1.16.0
• Component/s: Runtime / Network
• Labels: test-stability
• Assignee: Yuxin Tan
• Reporter: Leonard Xu
• Created: 03/Aug/22 09:40
• Updated: 08/Aug/22 03:16
• Resolved: 08/Aug/22 03:16

Description

switched from CANCELING to CANCELED.
2022-08-03 08:03:02,776 INFO  org.apache.flink.runtime.taskmanager.Task                    [] - Freeing task resources for MultipleInput[2212] -> Calc[2191] -> HashAggregate[2192] (8/8)#1 (cf5f33b100f0efb21b9ff8d27a78cd8e_d806bb3f5ea308ac3f1df304a96163b4_7_1).
2022-08-03 08:03:02,776 ERROR org.apache.flink.runtime.taskmanager.Task                    [] - Failed to release input gate for task MultipleInput[2212] -> Calc[2191] -> HashAggregate[2192] (8/8)#1.
org.apache.flink.shaded.netty4.io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at org.apache.flink.shaded.netty4.io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.runtime.io.network.buffer.NetworkBuffer.recycleBuffer(NetworkBuffer.java:156) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.runtime.io.network.buffer.ReadOnlySlicedNetworkBuffer.recycleBuffer(ReadOnlySlicedNetworkBuffer.java:123) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.runtime.io.network.buffer.CompositeBuffer.recycleBuffer(CompositeBuffer.java:70) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_332]
	at org.apache.flink.runtime.io.network.partition.SortMergeSubpartitionReader.releaseInternal(SortMergeSubpartitionReader.java:181) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.runtime.io.network.partition.SortMergeSubpartitionReader.releaseAllResources(SortMergeSubpartitionReader.java:163) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.runtime.io.network.partition.consumer.LocalInputChannel.releaseAllResources(LocalInputChannel.java:341) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.close(SingleInputGate.java:667) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.runtime.taskmanager.InputGateWithMetrics.close(InputGateWithMetrics.java:140) ~[flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.runtime.taskmanager.Task.closeAllInputGates(Task.java:1010) [flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.runtime.taskmanager.Task.releaseResources(Task.java:975) [flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:820) [flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:550) [flink-dist-1.16-SNAPSHOT.jar:1.16-SNAPSHOT]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_332]
2022-08-03 08:03:02,778 WARN  org.apache.flink.metrics.MetricGroup     

The failed CI link: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=39152&view=results

Issue Links

• is caused by: FLINK-28373 Read a full buffer of data per file IO read request for sort-shuffle