[GOBBLIN-684] Ensure buffered messages are flushed before close() in KafkaProducerPusher

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.15.0
• Fix Version/s: 0.15.0
• Component/s: gobblin-metrics
• Assignee: Sudarshan Vasudevan
• Reporter: Sudarshan Vasudevan
• Created: 15/Feb/19 00:31
• Updated: 20/Feb/19 14:56
• Resolved: 20/Feb/19 14:56

Description

Currently, when KafkaProducerPusher is closed, it invokes KafkaProducer#close(). However,close() only guarantees delivery of in-flight messages, not the messages in the producer buffer waiting to be sent out. This results in data loss.

The fix ensures that we call flush() before close(). As a result, any buffered messages are immediately pushed out and we block until the messages are acked. 

 

Issue Links

• links to: GitHub Pull Request #2556