After upgrading to Samza 1.2, the samza metrics high-watermark and messages-behind-high-watermark stopped updating. With a bit of digging, this was traced to our use of dot in our kafka topic names.
It looks like kafka refactored their metrics names significantly, and moved the topic name out of the metric name and into tags. When doing this, they added a replacement of '.' -> '_'. When Samza refactored to match this, the same replacement was not made, so the lookup of the "records-lag" metric in KafkaConsumerProxy fails to find a match.
The related commit on Samza is here:
The fix is a simple replacement added to the line above. Pull request coming shortly.