Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2230

Configuring elasticsearch-sink hostNames parameter in AWS

    XMLWordPrintableJSON

Details

    • Question
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • AWS, centos 6

    Description

      Hello,

      We are using flume elasticsearch-sink in AWS and for the hostNames parameter of the sink we use the A record of an internal ELB (Elastic Load Balancer). When we do a nslookup on the load balancer's hostname we get back a list of node IPs (in our case we have 3 elasticsearch nodes).

      config example

      a1.sinks.elasticsearch-sink.hostNames = componen-1RSEO3YX5OD3Z-729046292.eu-west-1.elb.amazonaws.com 
      

      This config works fine as long as the IPs of the elasticsearch nodes remain the same. If we restart one of our elasticsearch nodes, a new IP is assigned to it and flume stops being able to communicate with that node.

      In the source code of the elasticsearch-sink I can see that a list of InetSocketTransportAddress objects is created and this is probably the reason why flume stops working when we have an IP change and starts working only after a restart of the flume-ng-agent service.

      ElasticSearchSink.java

      serverAddresses[i] = new InetSocketTransportAddress(host, port);

      Questions:

      • Which is the suggested configuration for our case? Should we use static IPs for our elasticsearch nodes and then use a comma separated list of these IPs in flume configuration?
      • Would be possible to use the A record of the ELB in such a way that flume would always hit the A record to get one of the available IP addresses? Does this sound feasible and worth spending some time on submitting a patch?

      Regards,
      Nick

      Attachments

        Activity

          People

            Unassigned Unassigned
            nicktgr15 Nikolaos Tsipas
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: