Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2476

ContentBuilderUtil.appendField incorrectly manages json-like data

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: 1.5.0.1
    • Fix Version/s: 1.6.0
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      elasticsearch 1.1.0, elasticsearch 0.90.13

      Description

      There is a problem in org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer.getContentBuilder returns incorrect value, in case when Event contains well-formed json (possible as sub-string).

      Example:

      ElasticSearchDynamicSerializer ser = new ElasticSearchDynamicSerializer();
      Event event = new SimpleEvent();
      event.setBody("{\"true\":\"false\"}".getBytes());
      System.out.println(ser.getContentBuilder(event).string());
      //prints:
      //{"body":"org.elasticsearch.common.xcontent.XContentBuilder@31a5fdb9"}
      

      I tried to find origins of the problem, and found this chunk of code (flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java):

      public static void addComplexField(XContentBuilder builder, String fieldName,
            XContentType contentType, byte[] data) throws IOException {
          XContentParser parser = null;
          try {
            XContentBuilder tmp = jsonBuilder();
            parser = XContentFactory.xContent(contentType).createParser(data);
            parser.nextToken();
            tmp.copyCurrentStructure(parser);
            builder.field(fieldName, tmp); //here field (String, String) called, because there is no method(String,XContentBuilder) 
      //maybe tmp.string() should be here instead?
      

      It makes impossible to send any string, which contains json to elasticsearch sink using this serializer.

      Maybe I'm wrong, but what are benefits from decoding, then encoding chunk of json data?

      If it really needed, maybe it is possible to add some option to ElasticSearchDynamicSerializer which forces to use ContentBuilderUtil.addSimpleField instead of ContentBuilderUtil.appendField ?

      I can volunteer to create patch when will be decided what is the best way to avoid this issue.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                paliwalashish Ashish Paliwal
                Reporter:
                pavel.zalunin Pavel Zalunin
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: