[TWILL-204] AM should keep Kafka broker up until log consumers finish

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0, 0.10.0, 0.11.0
• Component/s: core
• Assignee: Unassigned
• Reporter: Martin Serrano
• Created: 14/Jan/17 19:10
• Updated: 21/Aug/17 16:23

Description

I'm encountering a situation on a fast machine where the Kafka log aggregation topic is not empty when the system shuts down. The scenario:

log consumer consumes all messages
consumer sleeps (500ms) due to empty queue
containers exit, posting final log messages about why
controller notices containers are down and terminates consumers.
consumer is interrupted from sleep and but has been canceled so it does not get the rest of the messages.

This scenario can be really confusing during development because an error may be missed (as in my case) if it falls into the final log messages. Before I file a ticket and fix this, I wanted to get some feedback. Looking at org.apache.twill.internal.kafka.client.SimpleKafkaConsumer it seems this behavior could be intentional given this log message (line 384):

LOG.debug("Unable to fetch messages on {}, kafka consumer service shutdown is in progress.", topicPart);

My opinion is that final messages logged by a container are likely to be critical in diagnosing errors and that twill should do whatever it can to forward them before shutting things down.