[FLUME-2661] Support timeout in ES Sink

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.5.1
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Ashish Paliwal
• Created: 08/Apr/15 03:40
• Updated: 21/Apr/15 14:28

Description

Credits to Otis, From ML

Hi,

I don't have a very good bug report, but I want to share one problem with
Flume we observed the other day. Specifically, with Elasticsearch Sink and
AWS ELB, but others may have this problem, too.

In short, ELB has this notion of a Spillover where it can reject requests
when it can't handle the load. Unfortunately, it doesn't actually signal
this by closing the connection, so Flume's connection remains open, I
believe in some java.net....... readSocket(....) method.

To work around this we had to add a socket timeout in our custom version of
Flume's Elasticsearch Sink.

I think it's worth fixing this in Flume and checking if other Sinks need
this sort of timeout.

HTH

Otis