Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41199

Streaming query metrics is broken with mixed-up usage of DSv1 streaming source and DSv2 streaming source

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.2, 3.3.1, 3.4.0
    • 3.4.0
    • Structured Streaming
    • None

    Description

      (It seems like a long standing issue. It probably applies to 2.x as well. I just marked the version line we did not EOL.)

      If a streaming query contains both DSv1 and DSv2 streaming sources together, it only collects metrics properly for DSv1 sources.

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kabhwan Jungtaek Lim Assign to me
            kabhwan Jungtaek Lim
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment