Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-4330

Add Kafka topics lag information on metrics and log

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backed up messages in Atlas: 

      It will also be useful to print the lag information periodically in atlas application.log on a frequency that can be made configurable (print lag every 1 hour, 5min or so).

      atlas.notification.consumer.topic.report.frequency.seconds=60

      2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
      2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
      

        Attachments

          Activity

            People

            • Assignee:
              sarath Sarath Subramanian
              Reporter:
              sarath Sarath Subramanian

              Dates

              • Created:
                Updated:

                Issue deployment