Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2089

ElasticsearchSink blocks and raises exceptions when event body has unexpected encoding

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4.0, 1.3.1
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Release Note:
      ElasticsearchSink now handles event bodies with unexpected encodings or parse failures by storing them as simple fields

      Description

      Detected by Allan Feid and documented on the user list http://mail-archives.apache.org/mod_mbox/flume-user/201306.mbox/%3CCAN94UWe6UvcOKT1S%2BXANC-sy0qFsxet3RJY9PVkj-eSfO5fk6Q%40mail.gmail.com%3E

      Steps:
      Send an event with the body as follows:
      foo¤data¤1371126476.436¤0.005¤555¤10.1.1.1¤HTTP/1.1¤GET¤http¤vhost¤/path/url¤¤-¤200¤
      referrer.com/search/?query=\x8D\x91\x89\xEF\x8Bc\x8E\x96\x93\xB0¤¤¤-

      Expected Results:
      The event is stored in elasticsearch.

      Actual Results:
      >> 10 Jun 2013 09:52:34,360 ERROR
      >> [SinkRunner-PollingRunner-DefaultSinkProcessor]
      >> (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver
      >> event. Exception follows.
      >> org.elasticsearch.common.jackson.dataformat.yaml.snakeyaml.error.YAMLException:
      >> java.io.CharConversionException: Invalid UTF-8 start byte 0xfc (at char
      >> #81, byte #-1)
      >> at
      >> org.elasticsearch.common.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:198)
      >> at
      >> org.elasticsearch.common.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.<init>(StreamReader.java:62)
      >> at
      >> org.elasticsearch.common.jackson.dataformat.yaml.YAMLParser.<init>(YAMLParser.java:147)
      >> at
      >> org.elasticsearch.common.jackson.dataformat.yaml.YAMLFactory._createParser(YAMLFactory.java:530)
      >> at
      >> org.elasticsearch.common.jackson.dataformat.yaml.YAMLFactory.createJsonParser(YAMLFactory.java:420)
      >> at
      >> org.elasticsearch.common.xcontent.yaml.YamlXContent.createParser(YamlXContent.java:83)
      >> at
      >> org.apache.flume.sink.elasticsearch.ContentBuilderUtil.addComplexField(ContentBuilderUtil.java:61)
      >> at
      >> org.apache.flume.sink.elasticsearch.ContentBuilderUtil.appendField(ContentBuilderUtil.java:47)
      >> at
      >> org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.appendBody(ElasticSearchLogStashEventSerializer.java:87)
      >> at
      >> org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer.getContentBuilder(ElasticSearchLogStashEventSerializer.java:79)
      >> at
      >> org.apache.flume.sink.elasticsearch.ElasticSearchSink.process(ElasticSearchSink.java:178)
      >> at
      >> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
      >> at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
      >> at java.lang.Thread.run(Thread.java:662)
      >> Caused by: java.io.CharConversionException: Invalid UTF-8 start byte 0xfc
      >> (at char #81, byte #-1)
      >> at
      >> org.elasticsearch.common.jackson.dataformat.yaml.UTF8Reader.reportInvalidInitial(UTF8Reader.java:395)
      >> at
      >> org.elasticsearch.common.jackson.dataformat.yaml.UTF8Reader.read(UTF8Reader.java:247)
      >> at
      >> org.elasticsearch.common.jackson.dataformat.yaml.UTF8Reader.read(UTF8Reader.java:157)
      >> at
      >> org.elasticsearch.common.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:182)
      >> ... 13 more

        Attachments

        1. flume-2089.diff
          3 kB
          Edward Sargisson

          Issue Links

            Activity

              People

              • Assignee:
                paliwalashish Ashish Paliwal
                Reporter:
                ejsarge Edward Sargisson
              • Votes:
                2 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: