Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-7388

Apache Drill Kafka Storage module fails to return results for partitions containing single offset record

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.16.0
    • Fix Version/s: 1.17.0
    • Component/s: None
    • Labels:

      Description

      If a partition only contains one record - e.g.

      [topicName=myTopic, partitionId=117, startOffset=0, endOffset=1]

      no data is returned.

      I fixed this locally with the following code change in contrib/storage-kafka :-

      git diff src/main/java/org/apache/drill/exec/store/kafka/KafkaRecordReader.java
      
      @@ -109,7 +109,7 @@ public class KafkaRecordReader extends AbstractRecordReader {
           currentMessageCount = 0;
       
           try {
      -      while (currentOffset < subScanSpec.getEndOffset() - 1 && msgItr.hasNext()) {
      +      while (currentOffset < subScanSpec.getEndOffset() && msgItr.hasNext()) {
               ConsumerRecord<byte[], byte[]> consumerRecord = msgItr.next();
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                arina Arina Ielchiieva
                Reporter:
                atlassdan daniel kelly
                Reviewer:
                Vova Vysotskyi
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: