Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36545

sparkstreaming input rate exceed spark.streaming.kafka.maxRatePerPartition

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Invalid
    • 2.4.5
    • None
    • DStreams, Spark Submit, YARN
    • None

    Description

       

      i have a spark streaming application with kafka . 
      Here are the parameters:

      kafka partition = 500

      batch time = 60 

      --conf spark.streaming.backpressure.enabled=true

      --conf spark.streaming.kafka.maxRatePerPartition=2500

      input size= 500 * 120 * 2500 = 75,000,000

      however input size become 160000000 after some batch 

      who can tell me reason

       

      Attachments

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            handong handong
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment