Flume
  1. Flume
  2. FLUME-2126

Problem in elasticsearch sink when the event body is a complex field

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.6.0
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      1.3.1 and 1.4

      Description

      I have found a bug in the elasticsearch sink, the problem is in the ContentBuilderUtil.addComplexField method, when it does builder.field(fieldName, tmp); the tmp object is taken as Object with the result of being serialized with the toString method in the XContentBuilder. In the end you get the object reference as content.

      The following change workaround the problem for me, the bad point is that it has to parse the content twice, I guess there is a better way to solve the problem but I am not an elasticsearch api expert.

      --- a/flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
      +++ b/flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
      @@ -61,7 +61,12 @@ public class ContentBuilderUtil {
             parser = XContentFactory.xContent(contentType).createParser(data);
             parser.nextToken();
             tmp.copyCurrentStructure(parser);
      -      builder.field(fieldName, tmp);
      +
      +      // if it is a valid structure then we include it
      +      parser = XContentFactory.xContent(contentType).createParser(data);
      +      parser.nextToken();
      +      builder.field(fieldName);
      +      builder.copyCurrentStructure(parser);
           } catch (JsonParseException ex) {
             // If we get an exception here the most likely cause is nested JSON that
             // can't be figured out in the body. At this point just push it through
      
      1. FLUME-2126-0.patch
        2 kB
        Ashish Paliwal

        Issue Links

          Activity

          Johny Rufus made changes -
          Fix Version/s v1.6.0 [ 12327047 ]
          Fix Version/s 1.6 [ 12329676 ]
          Francis made changes -
          Link This issue relates to FLUME-2649 [ FLUME-2649 ]
          Francis made changes -
          Link This issue relates to FLUME-2649 [ FLUME-2649 ]
          Francis made changes -
          Link This issue is related to FLUME-2649 [ FLUME-2649 ]
          Jarek Jarcec Cecho made changes -
          Fix Version/s 1.6 [ 12329676 ]
          Jarek Jarcec Cecho made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Ashish Paliwal made changes -
          Assignee Ashish Paliwal [ paliwalashish ]
          Hari Shreedharan made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hari Shreedharan made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Ashish Paliwal made changes -
          Link This issue is duplicated by FLUME-2476 [ FLUME-2476 ]
          Ashish Paliwal made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Ashish Paliwal made changes -
          Remote Link This issue links to "Review Request (Web Link)" [ 19050 ]
          Ashish Paliwal made changes -
          Attachment FLUME-2126-0.patch [ 12673264 ]
          Ashish Paliwal made changes -
          Field Original Value New Value
          Assignee Ashish Paliwal [ paliwalashish ]
          Massimo Paladin created issue -

            People

            • Assignee:
              Unassigned
              Reporter:
              Massimo Paladin
            • Votes:
              2 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development