Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-503

Lag gauge very slow to update for slow jobs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.9.0
    • Component/s: metrics
    • Labels:
      None
    • Environment:

      Mac OS X, Oracle Java 7, ProcessJobFactory

      Description

      For slow jobs, the KafkaSystemConsumerMetrics.%s-%s-messages-behind-high-watermark) gauge does not get updated very often.

      To reproduce:

      • Create a job that processes one message and sleeps for 5 seconds
      • Create it's input topic but do not populate it yet
      • Start the job
      • Load 1000s of messages to it's input topic. You can keep adding messages with a "watch -n 1 <kafka console producer command>"

      What happens:

      • Run jconsole to view the JMX metrics
      • The %s-%s-messages-behind-high-watermark gauge will stay at 0 for a LONG time (~10 minutes?) before finally updating.

      What should happen:

      • The gauge should get updated at a reasonable interval (a least every few seconds)

      I think what's happening is that the BrokerProxy only updates the high watermark when a consumer is ready for more messages. When the job is so slow, this rarely happens to the metric doesn't get updated.

        Attachments

        1. SAMZA-503.3.patch
          6 kB
          Yan Fang
        2. SAMZA-503.2.patch
          6 kB
          Yan Fang
        3. SAMZA-503.2.patch
          5 kB
          Yan Fang
        4. SAMZA-503.1.patch
          3 kB
          Yan Fang
        5. SAMZA-503.patch
          1 kB
          Yan Fang

          Issue Links

            Activity

              People

              • Assignee:
                closeuris Yan Fang
                Reporter:
                theduderog Roger Hoover
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: