[HUDI-3373] Add zero value counter metrics because of Prometheus PushGateway stale metrics problem

• Type: Task
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: metrics
• Labels: pull-request-available
• Epic Link: Enhance metrics
• Assignee: Unassigned
• Reporter: Vinish Reddy

Reviewers

sivabalan narayanan

• Created: 07/Feb/22 17:05
• Updated: 03/Mar/22 15:34
• Resolved: 03/Mar/22 15:34

Description

Prometheus Pushgateway never forgets series pushed to it and will expose them to Prometheus forever unless those series are manually deleted via the Pushgateway's API. When deltastreamer finds that there are no new records to be read from the source since the last checkpoint it doesn't emit a metric because the write path is not triggered and leads to a graph like below (a straight line with constant value as metrics emitted) in case of PushGateway metrics reporter. For other metrics reporter like DataDog etc. it would be a null value and no data will be shown in the graph.

22/02/07 11:23:10 INFO DeltaSync: No new data, source checkpoint has not changed. Nothing to commit. Old checkpoint=(Option{val=20220207062847034}). New Checkpoint=(20220207062847034)

 

To give a better representation of the actual metrics for Prometheus PushGateway, the counter metrics have to be updated with zero value so that it's better than having stale data which was ingested as part of the last checkpoint.

 

{{}}

Issue Links

• links to: GitHub Pull Request #4760