[STORM-1698] Asynchronous MetricsConsumerBolt

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0, 2.0.0
• Fix Version/s: 2.0.0, 1.1.0
• Component/s: storm-core
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 08/Apr/16 05:49
• Updated: 13/Jun/16 04:21
• Resolved: 13/Jun/16 04:21

Description

Currently MetricsConsumerBolt is delegating MetricsConsumer to handle data points via synchronous manner.

When MetricsConsumer cannot keep up, it will trigger backpressure when (queue size + overflow buffer size) reaches high watermark, which incurs slowing down the topology in result.

Slowing down Itself is not a problem because that’s what backpressure is for. The actual problem is that backpressure only throttles spout, not metrics. If MetricsConsumerBolt cannot keep up with incoming tuples, backpressure never ends and topology just hangs. If we turn off backpressure, we have unbounded queue and worker could throw OOME eventually.

Making MetricsConsumerBolt asynchronous can resolve this issue. One downside of making it async is that it's hard to see that MetricsConsumerBolt is keeping up now. (capacity will be always around 0)
I don't have an idea for now but I think it's still better than current.

Before making consensus about huge change of metrics, I'd love to improve current metrics without breaking backward compatible manner. It could be applied to 1.x-branch, and even 0.10.x-branch.

Issue Links

• Is contained by: STORM-1700 Introduce 'whitelist' / 'blacklist' option to MetricsConsumer
• requires: STORM-1832 Consistently slow metrics consumer triggers backpressure which will be never back to normal