Affects Version/s: 1.15.0, 1.16.0
Fix Version/s: None
Component/s: Storage - Kafka
When we try to query Mapr-streams(similar to Apache Kafka) topic using Kafka plugin, we see the below timeout being thrown.
The other interesting behavior is that:
1) Even after increasing the timeout value to 50secs, the drill query failed after the execution time crossed 50 secs(~51 secs).
This pattern continued to whatever value we increased. For ex, after increasing the timeout to 100secs, query failed with above error after 101 secs of execution time.
The user is using Drill 1.15.
I tried to reproduce this on my test cluster. But this was not consistently reproducing in the test cluster. Whereas in the client's cluster, we were able to reproduce this behavior consistently. We collected the logs. But they have very little info on what's happening.
I believe it is now essential to know how(and why) the timeout parameter "store.kafka.poll.timeout" is related to the Query execution time to understand this bug.
I also have few more questions where i couldn't find much documentation.
1) Is there a one-to-one mapping between number of partitions of a topic to minor fragments of a query? For ex, If a given topic(t) has say 20 partitions, then will the query most likely have 20 minor fragments, other parameters being fairly sized.
2) Does each drillbit in the cluster equivalent to a Kafka-Consumer and all such drillbits of a cluster treated as part of a consumer group?