[FLINK-19925] Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 1.12.0
• Fix Version/s: 1.14.0, 1.12.5
• Component/s: Runtime / Network
• Labels: test-stability
• Assignee: Unassigned
• Reporter: godfrey he
• Created: 02/Nov/20 10:11
• Updated: 22/Jun/21 13:55
• Resolved: 19/May/21 07:26

Description

Errors$NativeIoException will occur sometime when we run TPCDS based on master, the full exception stack is

Caused by: org.apache.flink.runtime.io.network.netty.exception.LocalTransportException: readAddress(..) failed: Connection reset by peer (connection to 'xxx')
	at org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.exceptionCaught(CreditBasedPartitionRequestClientHandler.java:173) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.handleReadException(AbstractEpollStreamChannel.java:728) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:818) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
	at java.lang.Thread.run(Thread.java:834) ~[?:1.8.0_102]
Caused by: org.apache.flink.shaded.netty4.io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer

Issue Links

• relates to: FLINK-21416 FileBufferReaderITCase.testSequentialReading fails on azure