Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.0.0, 1.1.2, 1.0.6, 1.2.1
Description
A topology that consumes from two different Kafka clusters: 0.10.1.1 and 0.10.2.1.
Spouts consuming from 0.10.2.1 have a low lag (and regularly commit offsets)
The Spout that consumes from 0.10.1.1 exhibits either:
1- Unknown lag
2- Lag that increments as the Spout reads messages from Kafka
In DEBUG, Offset manager logs: "topic-partition has NO offsets ready to be committed", despite continuing to consume messages.
Several configuration tweaks were tried, including setting maxRetries to 1, in case messages with a lower offset were being retried (logs didn't show it, though)
offsetCommitPeriodMs was also lowered to no avail.
The only configuration that works is to have ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG=true, but this is undesired since we lose processing guarantees.
