[SAMZA-503] Lag gauge very slow to update for slow jobs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 0.9.0
• Component/s: metrics

Environment

Mac OS X, Oracle Java 7, ProcessJobFactory

• Assignee: Yan Fang
• Reporter: Roger Hoover
• Created: 06/Jan/15 06:27
• Updated: 31/Jan/15 07:01
• Resolved: 30/Jan/15 19:08

Description

For slow jobs, the KafkaSystemConsumerMetrics.%s-%s-messages-behind-high-watermark) gauge does not get updated very often.

To reproduce:

• Create a job that processes one message and sleeps for 5 seconds
• Create it's input topic but do not populate it yet
• Start the job
• Load 1000s of messages to it's input topic. You can keep adding messages with a "watch -n 1 <kafka console producer command>"

What happens:

• Run jconsole to view the JMX metrics
• The %s-%s-messages-behind-high-watermark gauge will stay at 0 for a LONG time (~10 minutes?) before finally updating.

What should happen:

• The gauge should get updated at a reasonable interval (a least every few seconds)

I think what's happening is that the BrokerProxy only updates the high watermark when a consumer is ready for more messages. When the job is so slow, this rarely happens to the metric doesn't get updated.

Issue Links

• is related to: SAMZA-539 Add generic lag metric from high watermark - last processed message
• is related to: SAMZA-540 Expose latency related metrics in OffsetManager