The failure was caused by the following sequence of steps -
- Test queries for available hosts where a segment is located and gets the location of kafka task.
- Kafka task hands over the data and finishes
- Now the scan query is sent to the kafka task, but the task has already completed and will fail.
https://issues.apache.org/jira/browse/HIVE-20349 fixes this issue by retrying the broker in this case.
One more cause of failure was the latestOffsets and minimumLag not reported when there is no task.
This patch masks those two values also. Query results are verified to ensure that there is no lag.