[KYLIN-3942] Rea-time OLAP don't support multi-level json event

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: v3.0.0-alpha
• Fix Version/s: v3.0.0-alpha2
• Component/s: Real-time Streaming
• Assignee: Xiaoxiang Yu
• Reporter: Xiaoxiang Yu
• Created: 09/Apr/19 10:02
• Updated: 21/Jan/20 07:43
• Resolved: 18/Apr/19 05:48

Description

Currently real-time OLAP didn't support  multi-level json event.

For example,if I have a kafka multi-level json event like this:

{"country":"JAPAN","amount":13.075058425023922,"qty":8,"currency":"USD","order_time":1554801950882,"category":"ELECTRONIC","device":"Andriod","user":{"gender":"Female","id":"7a0cfa5e-bbaa-79ef-1a38-e06f02c85fcb","first_name":"unknown","age":16}}

 

Receiver will throw exception like this and discard that event:

 

2019-04-09 09:46:09,878 ERROR [StreamingV2Cube_channel] kafka.TimedJsonStreamParser:107 : error
com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token
at [Source: (String)"\{"country":"US","amount":14.498498222823619,"qty":1,"currency":"USD","order_time":1554803169876,"category":"Other","device":"Other","user":{"gender":"Female","id":"0736b41a-9ae7-9b4a-a124-f74436d3eb41","first_name":"unknown","age":26}}"; line: 1, column: 140] (through reference chain: java.util.HashMap["user"])
at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092)
at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:63)
at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:527)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:364)
at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3030)
at org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:79)
at org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:54)
at org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.nextEvent(KafkaConnector.java:110)
at org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:93)
at java.lang.Thread.run(Thread.java:748)

Issue Links

• links to: GitHub Pull Request #605