For slow jobs, the KafkaSystemConsumerMetrics.%s-%s-messages-behind-high-watermark) gauge does not get updated very often.
- Create a job that processes one message and sleeps for 5 seconds
- Create it's input topic but do not populate it yet
- Start the job
- Load 1000s of messages to it's input topic. You can keep adding messages with a "watch -n 1 <kafka console producer command>"
- Run jconsole to view the JMX metrics
- The %s-%s-messages-behind-high-watermark gauge will stay at 0 for a LONG time (~10 minutes?) before finally updating.
What should happen:
- The gauge should get updated at a reasonable interval (a least every few seconds)
I think what's happening is that the BrokerProxy only updates the high watermark when a consumer is ready for more messages. When the job is so slow, this rarely happens to the metric doesn't get updated.