[DRILL-7388] Apache Drill Kafka Storage module fails to return results for partitions containing single offset record

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.0
• Fix Version/s: 1.17.0
• Labels: ready-to-commit
• Assignee: Arina Ielchiieva
• Reporter: daniel kelly
• Reviewer: Vova Vysotskyi
• Created: 25/Sep/19 13:27
• Updated: 20/Nov/19 11:46
• Resolved: 20/Nov/19 11:46

Description

If a partition only contains one record - e.g.

[topicName=myTopic, partitionId=117, startOffset=0, endOffset=1]

no data is returned.

I fixed this locally with the following code change in contrib/storage-kafka :-

git diff src/main/java/org/apache/drill/exec/store/kafka/KafkaRecordReader.java

@@ -109,7 +109,7 @@ public class KafkaRecordReader extends AbstractRecordReader {
     currentMessageCount = 0;
 
     try {
-      while (currentOffset < subScanSpec.getEndOffset() - 1 && msgItr.hasNext()) {
+      while (currentOffset < subScanSpec.getEndOffset() && msgItr.hasNext()) {
         ConsumerRecord<byte[], byte[]> consumerRecord = msgItr.next();

Issue Links

• incorporates: DRILL-6723 Kafka reader fails on malformed JSON
• incorporates: DRILL-7290 “Failed to construct kafka consumer” using Apache Drill
• incorporates: DRILL-6739 Update Kafka libs to 2.0.0+ version
• links to: GitHub Pull Request #1901