[STORM-2786] Ackers leak tracking info on failure and lots of other cases.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.9.1-incubating, 0.10.0, 1.0.0, 2.0.0
• Fix Version/s: 2.0.0, 1.2.0, 1.1.2, 1.0.6
• Component/s: storm-client, storm-core
• Labels: pull-request-available
• Assignee: Robert Joseph Evans
• Reporter: Robert Joseph Evans
• Created: 23/Oct/17 12:21
• Updated: 25/Oct/17 22:05
• Resolved: 25/Oct/17 22:05

Description

Over the weekend we had an incident where ackers were running out of memory at a really scary rate. It turns out that they were having a lot of failures, for an unrelated reason, but each of the failures were resulting in tuple tracking being lost because...

We don't send ticks to any system components ever...

https://github.com/apache/storm/blob/124acb92dff04a57b530ab4d95a698abc8ff46d9/storm-client/src/jvm/org/apache/storm/executor/Executor.java#L384

and ackers are system components.

So the tracking map was never rotated and all failed tuples

https://github.com/apache/storm/blob/124acb92dff04a57b530ab4d95a698abc8ff46d9/storm-client/src/jvm/org/apache/storm/daemon/Acker.java#L97-L103

Were never deleted from the map.

This leak eventually made the ackers crash, and when they came back up the other components kept blasting them with messages that would never be fully acked which also leaked because of the tick problem.

Looking back this has been in every release since 0.9.1-incubating. It appears to have been introduced by https://github.com/apache/storm/commit/483ce454a3b2cd31b5d1c34e9365346459b358a8

So every apache release has this problem (which is the only reason I have not marked this as a blocker, because apparently it is not so bad that anyone has noticed in the past 4 years).

Issue Links

• links to: GitHub Pull Request #2382
• links to: GitHub Pull Request #2383