Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2220

ElasticSearch sink - duplicate fields in indexed document

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Patch Available
    • Minor
    • Resolution: Unresolved
    • 1.4.0
    • None
    • None

    Description

      The default serializer for the ElasticSearch sink (ElasticSearchLogStashEventSerializer) duplicates fields that are mapped to default logstash fields.
      For instance timestamp, source, host. Those appear both as logstash fields ("@timestamp", "@source_host" etc.), and both as fields under the @fields ("@fields.timestamp", "@fields.host").
      When inserting a field from the headers as a logstash system field it should be removed from the dictionary so it wouldn't get written again under the "@fields" field.

      Attachments

        1. FLUME-2220.patch
          4 kB
          Rotem Hermon

        Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. FLUME-2099 Add serializer to ElasticSearchSink for new logstash v1 format

          • Major - Major loss of function.
          • Open

          Activity

            People

              rore Rotem Hermon
              rore Rotem Hermon
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: