Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-4330

Add Kafka topics lag information on metrics and log

    XMLWordPrintableJSON

    Details

      Description

      In metrics page, in addition to other Notification details. Lag information of kafka consumer topics will be useful to check the current backed up messages in Atlas: 

      It will also be useful to print the lag information periodically in atlas application.log on a frequency that can be made configurable (print lag every 1 hour, 5min or so).

      atlas.notification.consumer.topic.report.frequency.seconds=60

      2021-06-07 21:56:06,121 INFO - NotificationProcessingStats: topic=ATLAS_SPARK_HOOK-0, currentOffset=18, endOffset=18, lag=0 
      2021-06-07 21:56:09,105 INFO - NotificationProcessingStats: topic=ATLAS_HOOK-0, currentOffset=1118, endOffset=1118, lag=0
      

        Attachments

        1. image-2021-06-07-15-03-28-166.png
          21 kB
          Sarath Subramanian

          Issue Links

            Activity

              People

              • Assignee:
                sarath Sarath Subramanian
                Reporter:
                sarath Sarath Subramanian
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: