[IGNITE-14331] Possible distributed race related to a data streamer flushing leading to a thread being stuck forever trying to close the streamer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.10
• Fix Version/s: 2.11
• Component/s: streaming
• Release Note: Fixed an issue when Data Streamer flushing could hang due to a connectivity problem.
• Ignite Flags: Release Notes Required
• Assignee: Vyacheslav Koptilin
• Reporter: Vladimir Pligin
• Reviewer: Vladislav Pyatkov
• Created: 17/Mar/21 13:06
• Updated: 01/Jul/21 10:03
• Resolved: 15/Apr/21 19:31

Description

It seems that a streamer could stuck forever flushing internal buffers on a client side.

It will stay in a busy-loop forever hoping on remapping but it's possible that it won't happen for example in case of long GC pauses on server(s) and long timeouts.

It that case a streamer would be trapped inside this loop.

Stack trace snippet:

java.lang.Thread.State: RUNNABLE        at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.flush(DataStreamerImpl.java:1706)        at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.doFlush(DataStreamerImpl.java:1170)        at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.closeEx(DataStreamerImpl.java:1365)        at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.closeEx(DataStreamerImpl.java:1323)        at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.close(DataStreamerImpl.java:1311)        at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.close(DataStreamerImpl.java:1415)

 

It becomes possible when a 
IgniteSpiOperationTimeoutException
is being thrown from 
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic()
 

Issue Links

• links to: GitHub Pull Request #8955