[DRILL-6723] Kafka reader fails on malformed JSON

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.17.0
• Component/s: Storage - JSON

Environment

java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

$ bin/drill-embedded
Aug 30, 2018 5:29:08 PM org.glassfish.jersey.server.ApplicationHandler initialize
INFO: Initiating Jersey application, version Jersey: 2.8 2014-04-29 01:25:26...
apache drill 1.14.0

 

• Assignee: Arina Ielchiieva
• Reporter: Matt Keranen
• Created: 30/Aug/18 21:29
• Updated: 20/Nov/19 11:45
• Resolved: 20/Nov/19 11:45

Description

With a Kakfa topic where the first message is a simple string "test", and store.json.reader.skip_invalid_records = true, a SELECT * FROM topic still fails following error.

In addition, using an OFFSET does not appear to allow the bad messages to be bypassed. Same error occurs on the first message.

0: jdbc:drill:zk=local> select * from kafka.`logs` limit 5;
Error: DATA_READ ERROR: Failure while reading messages from kafka. Recordreader was at record: 1

Not a JSON Object: "TEST"
Fragment 0:0

[Error Id: 965d7a69-3d77-4a11-9613-3892a95c4a63 on x.x.x.x:31010] (state=,code=0)

Description:
New option store.kafka.reader.skip_invalid_records will be introduced to cover this case. Default is false.

Issue Links

• is part of: DRILL-7388 Apache Drill Kafka Storage module fails to return results for partitions containing single offset record
• is related to: DRILL-4653 Malformed JSON should not stop the entire query from progressing