[FLUME-2476] ContentBuilderUtil.appendField incorrectly manages json-like data

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: 1.5.0.1
• Fix Version/s: 1.6.0
• Component/s: Sinks+Sources

Environment

elasticsearch 1.1.0, elasticsearch 0.90.13

• Assignee: Ashish Paliwal
• Reporter: Pavel Zalunin
• Created: 24/Sep/14 21:13
• Updated: 16/Jun/16 19:01
• Resolved: 15/Oct/14 13:59

Description

There is a problem in org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer.getContentBuilder returns incorrect value, in case when Event contains well-formed json (possible as sub-string).

Example:

ElasticSearchDynamicSerializer ser = new ElasticSearchDynamicSerializer();
Event event = new SimpleEvent();
event.setBody("{\"true\":\"false\"}".getBytes());
System.out.println(ser.getContentBuilder(event).string());
//prints:
//{"body":"org.elasticsearch.common.xcontent.XContentBuilder@31a5fdb9"}

I tried to find origins of the problem, and found this chunk of code (flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java):

public static void addComplexField(XContentBuilder builder, String fieldName,
      XContentType contentType, byte[] data) throws IOException {
    XContentParser parser = null;
    try {
      XContentBuilder tmp = jsonBuilder();
      parser = XContentFactory.xContent(contentType).createParser(data);
      parser.nextToken();
      tmp.copyCurrentStructure(parser);
      builder.field(fieldName, tmp); //here field (String, String) called, because there is no method(String,XContentBuilder) 
//maybe tmp.string() should be here instead?

It makes impossible to send any string, which contains json to elasticsearch sink using this serializer.

Maybe I'm wrong, but what are benefits from decoding, then encoding chunk of json data?

If it really needed, maybe it is possible to add some option to ElasticSearchDynamicSerializer which forces to use ContentBuilderUtil.addSimpleField instead of ContentBuilderUtil.appendField ?

I can volunteer to create patch when will be decided what is the best way to avoid this issue.

Issue Links

• duplicates: FLUME-2126 Problem in elasticsearch sink when the event body is a complex field