Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6723

Kafka reader fails on malformed JSON

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.14.0
    • Fix Version/s: 1.17.0
    • Component/s: Storage - JSON
    • Labels:
      None
    • Environment:

      Description

      With a Kakfa topic where the first message is a simple string "test", and store.json.reader.skip_invalid_records = true, a SELECT * FROM topic still fails following error.

      In addition, using an OFFSET does not appear to allow the bad messages to be bypassed. Same error occurs on the first message.

      0: jdbc:drill:zk=local> select * from kafka.`logs` limit 5;
      Error: DATA_READ ERROR: Failure while reading messages from kafka. Recordreader was at record: 1
      
      Not a JSON Object: "TEST"
      Fragment 0:0
      
      [Error Id: 965d7a69-3d77-4a11-9613-3892a95c4a63 on x.x.x.x:31010] (state=,code=0)
      

      Description:
      New option store.kafka.reader.skip_invalid_records will be introduced to cover this case. Default is false.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                arina Arina Ielchiieva
                Reporter:
                mattk Matt Keranen
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: