[KUDU-2078] Flume sink fails to write if the size in bytes of the batch exceeds the client session's manual flush buffer size

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Fix Version/s: 1.5.0
• Component/s: flume-sink
• Assignee: William Berkeley
• Reporter: William Berkeley
• Created: 24/Jul/17 18:46
• Updated: 02/Oct/17 19:08
• Resolved: 02/Oct/17 19:08

Description

The Flume sink uses MANUAL_FLUSH mode. If a set of operations is too large for the flush buffer, it will fail-- potentially over and over, if the channel supports retrying the transaction.

We should be able to fix this by using AUTO_FLUSH_BACKGROUND, adjusting the buffer size, or returning a more useful error so that users can reconfigure their sink's batch size (though that size is in # of ops, not bytes).