Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2126

Problem in elasticsearch sink when the event body is a complex field

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.6.0
    • Sinks+Sources
    • None
    • 1.3.1 and 1.4

    Description

      I have found a bug in the elasticsearch sink, the problem is in the ContentBuilderUtil.addComplexField method, when it does builder.field(fieldName, tmp); the tmp object is taken as Object with the result of being serialized with the toString method in the XContentBuilder. In the end you get the object reference as content.

      The following change workaround the problem for me, the bad point is that it has to parse the content twice, I guess there is a better way to solve the problem but I am not an elasticsearch api expert.

      --- a/flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
      +++ b/flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
      @@ -61,7 +61,12 @@ public class ContentBuilderUtil {
             parser = XContentFactory.xContent(contentType).createParser(data);
             parser.nextToken();
             tmp.copyCurrentStructure(parser);
      -      builder.field(fieldName, tmp);
      +
      +      // if it is a valid structure then we include it
      +      parser = XContentFactory.xContent(contentType).createParser(data);
      +      parser.nextToken();
      +      builder.field(fieldName);
      +      builder.copyCurrentStructure(parser);
           } catch (JsonParseException ex) {
             // If we get an exception here the most likely cause is nested JSON that
             // can't be figured out in the body. At this point just push it through
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            mpaladin Massimo Paladin
            Votes:
            2 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment