[METRON-1897] Warn About Max Spout Pending Exceeded in Profiler Topology

• Type: Improvement
• Status: To Do
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Nick Allen
• Created: 27/Nov/18 17:42
• Updated: 27/Nov/18 17:43

Description

If the user sets the max spout pending value too low for the Profiler topology, it can cause the topology to stall and stop consuming messages.

This is an annoying issue to debug because I am unable to find anything logged by Storm indicating that the max spout pending limit is breached. When this limit is exceeded it just appears that the KafkaSpout stops consuming messages from Kafka and everything else would indicate the topology is healthy.

The Profiler topology is the only Metron topology that uses Storm's windowing support. Storm will queue up messages for the window (by default 30 seconds) and process each of the tuples in the window as one unit. This causes Storm to need to "hang on" to tuples longer than the other topologies and this requires a greater max spout pending value.

Ideally, we need to make it more obvious that exceeding the max spout pending value in the Profiler topology is causing the KafkaSpout to stop consuming messages.  

Issue Links

• relates to: METRON-1888 Default Topology Settings in MPack Cause Profiler to Stall