[STORM-329] Fix cascading Storm failure by improving reconnection strategy and buffering messages

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.2-incubating, 0.9.3
• Fix Version/s: 0.10.0, 0.9.4
• Component/s: storm-core
• Labels: Netty
• Assignee: Michael G. Noll
• Reporter: Sean Zhong
• Created: 21/May/14 01:47
• Updated: 05/Oct/15 02:12
• Resolved: 25/Feb/15 22:14

Description

Note: The original title of this ticket was: "Add Option to Config Message handling strategy when connection timeout".

This is to address a concern brought up during the work at STORM-297:

Robert Joseph Evans wrote: Your logic makes since to me on why these calls are blocking. My biggest concern around the blocking is in the case of a worker crashing. If a single worker crashes this can block the entire topology from executing until that worker comes back up. In some cases I can see that being something that you would want. In other cases I can see speed being the primary concern and some users would like to get partial data fast, rather then accurate data later.

Could we make it configurable on a follow up JIRA where we can have a max limit to the buffering that is allowed, before we block, or throw data away (which is what zeromq does)?

If some worker crash suddenly, how to handle the message which was supposed to be delivered to the worker?

1. Should we buffer all message infinitely?
2. Should we block the message sending until the connection is resumed?
3. Should we config a buffer limit, try to buffer the message first, if the limit is met, then block?
4. Should we neither block, nor buffer too much, but choose to drop the messages, and use the built-in storm failover mechanism?

Issue Links

• contains: STORM-404 Worker on one machine crashes due to a failure of another worker on another machine
• is related to: STORM-297 Storm Performance cannot be scaled up by adding more CPU cores
• is related to: STORM-547 Build Problem(s)
• relates to: STORM-510 Netty messaging client blocks transfer thread on reconnect
• relates to: STORM-677 Maximum retries strategy may cause data loss
• relates to: STORM-350 Update disruptor to latest version (3.3.2)

Sub-Tasks

• Worker on one machine crashes due to a failure of another worker on another machine: Resolved
• Netty messaging client blocks transfer thread on reconnect: Closed