Flume
  1. Flume
  2. FLUME-2230

Configuring elasticsearch-sink hostNames parameter in AWS

    Details

    • Type: Question Question
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Environment:

      AWS, centos 6

      Description

      Hello,

      We are using flume elasticsearch-sink in AWS and for the hostNames parameter of the sink we use the A record of an internal ELB (Elastic Load Balancer). When we do a nslookup on the load balancer's hostname we get back a list of node IPs (in our case we have 3 elasticsearch nodes).

      config example

      a1.sinks.elasticsearch-sink.hostNames = componen-1RSEO3YX5OD3Z-729046292.eu-west-1.elb.amazonaws.com 
      

      This config works fine as long as the IPs of the elasticsearch nodes remain the same. If we restart one of our elasticsearch nodes, a new IP is assigned to it and flume stops being able to communicate with that node.

      In the source code of the elasticsearch-sink I can see that a list of InetSocketTransportAddress objects is created and this is probably the reason why flume stops working when we have an IP change and starts working only after a restart of the flume-ng-agent service.

      ElasticSearchSink.java

      serverAddresses[i] = new InetSocketTransportAddress(host, port);

      Questions:

      • Which is the suggested configuration for our case? Should we use static IPs for our elasticsearch nodes and then use a comma separated list of these IPs in flume configuration?
      • Would be possible to use the A record of the ELB in such a way that flume would always hit the A record to get one of the available IP addresses? Does this sound feasible and worth spending some time on submitting a patch?

      Regards,
      Nick

        Activity

        There are no comments yet on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Nikolaos Tsipas
          • Votes:
            2 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development