Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3942

Rea-time OLAP don't support multi-level json event

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: v3.0.0-alpha
    • Fix Version/s: v3.0.0-alpha2
    • Component/s: Real-time Streaming
    • Labels:
      None

      Description

      Currently real-time OLAP didn't support  multi-level json event.

      For example,if I have a kafka multi-level json event like this:

      {"country":"JAPAN","amount":13.075058425023922,"qty":8,"currency":"USD","order_time":1554801950882,"category":"ELECTRONIC","device":"Andriod","user":{"gender":"Female","id":"7a0cfa5e-bbaa-79ef-1a38-e06f02c85fcb","first_name":"unknown","age":16}}

       

      Receiver will throw exception like this and discard that event:

       

      2019-04-09 09:46:09,878 ERROR [StreamingV2Cube_channel] kafka.TimedJsonStreamParser:107 : error
      com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.lang.String` out of START_OBJECT token
      at [Source: (String)"\{"country":"US","amount":14.498498222823619,"qty":1,"currency":"USD","order_time":1554803169876,"category":"Other","device":"Other","user":{"gender":"Female","id":"0736b41a-9ae7-9b4a-a124-f74436d3eb41","first_name":"unknown","age":26}}"; line: 1, column: 140] (through reference chain: java.util.HashMap["user"])
      at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:63)
      at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1342)
      at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1138)
      at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1092)
      at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:63)
      at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:10)
      at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:527)
      at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:364)
      at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:29)
      at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4001)
      at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3030)
      at org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:79)
      at org.apache.kylin.stream.source.kafka.TimedJsonStreamParser.parse(TimedJsonStreamParser.java:54)
      at org.apache.kylin.stream.source.kafka.consumer.KafkaConnector.nextEvent(KafkaConnector.java:110)
      at org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:93)
      at java.lang.Thread.run(Thread.java:748)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                hit_lacus Xiaoxiang Yu
                Reporter:
                hit_lacus Xiaoxiang Yu
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: