Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-2295

records-lag based topic metrics not working for topics with periods (dot, '.') in them

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2
    • Fix Version/s: 1.3
    • Component/s: kafka
    • Labels:
      None

      Description

      After upgrading to Samza 1.2, the samza metrics high-watermark and messages-behind-high-watermark stopped updating. With a bit of digging, this was traced to our use of dot in our kafka topic names.

      It looks like kafka refactored their metrics names significantly, and moved the topic name out of the metric name and into tags. When doing this, they added a replacement of '.' -> '_'. When Samza refactored to match this, the same replacement was not made, so the lookup of the "records-lag" metric in KafkaConsumerProxy fails to find a match.

      The commit on kafka is here:
      https://github.com/apache/kafka/commit/5d81639907869ce7355c40d2bac176a655e52074#diff-b45245913eaae46aa847d2615d62cde0R1331

       

      The related commit on Samza is here:
      https://github.com/apache/samza/commit/01c06055057e8c260e73bf3f7db7b9eb53af3745#diff-b56db493dfdd2fd2d471f6df342fe661R400

       

      The fix is a simple replacement added to the line above. Pull request coming shortly.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bharathkk Bharath Kumarasubramanian
                Reporter:
                thunderstumpges Thunder Stumpges
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m