Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34854

Report metrics for streaming source through progress reporter with Kafka source use-case

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • SQL
    • None

    Description

      Report Metrics for streaming source through progress reporter to users.

      Add kafka micro batch streaming use-case to report stats of # of offsets for the current offset falling behind the latest.

       

      SPARK-34366 and SPARK-34297 report metrics in spark ui, but this issue reports metrics through progress report, making it available to users via listener.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            yijia.cui Yijia Cui
            yijia.cui Yijia Cui
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment